Bayesian experimental design

Bayesian experimental design provides a general probability-theoretical framework from which other theories on experimental design can be derived. It is based on Bayesian inference to interpret the observations/data acquired during the experiment. This allows accounting for both any prior knowledge on the parameters to be determined as well as uncertainties in observations.

The theory of Bayesian experimental design is to a certain extent based on the theory for making optimal decisions under uncertainty. The aim when designing an experiment is to maximize the expected utility of the experiment outcome. The utility is most commonly defined in terms of a measure of the accuracy of the information provided by the experiment (e.g. the Shannon information or the negative variance), but may also involve factors such as the financial cost of performing the experiment. What will be the optimal experiment design depends on the particular utility criterion chosen.

Contents

Relations to more specialized optimal design theory

Linear theory

If the model is linear, the prior probability density function (PDF) is homogeneous and observational errors are normally distributed, the theory simplifies to the classical optimal experimental design theory.

Approximate normality

In numerous publications on Bayesian experimental design, it is (often implicitly) assumed that all posterior PDFs will be approximately normal. This allows for the expected utility to be calculated using linear theory, averaging over the space of model parameters, an approach reviewed in Chaloner & Verdinelli (1995). Caution must however be taken when applying this method, since approximate normality of all possible posteriors is difficult to verify, even in cases of normal observational errors and uniform prior PDF.

Mathematical formulation

Notation
\theta\, parameters to be determined
y\, observation or data
\xi\, design
p(y|\theta,\xi)\, PDF for making observation y, given parameter values \theta and design \xi
p(\theta)\, prior PDF
p(y|\xi)\, marginal PDF in observation space
p(\theta | y, \xi)\,    posterior PDF
U(\xi)\,    utility of the design \xi
U(y, \xi)\,    utility of the experiment outcome after observation y with design \xi

Given a vector \theta of parameters to determine, a prior PDF p(\theta) over those parameters and a PDF p(y|\theta,\xi) for making observation y, given parameter values \theta and an experiment design \xi, the posterior PDF can be calculated using Bayes' theorem

p(\theta | y, \xi) = \frac{p(y | \theta, \xi) p(\theta)}{p(y | \xi)} \, ,

where p(y|\xi) is the marginal probability density in observation space

p(y|\xi) = \int{p(\theta)p(y|\theta,\xi)d\theta}\, .

The expected utility of an experiment with design \xi can then be defined

U(\xi)=\int{p(y|\xi)U(y,\xi)dy}\, ,

where U(y,\xi) is some real-valued functional of the posterior PDF p(\theta | y, \xi) after making observation y using an experiment design \xi.

Gain in Shannon information as utility

If the utility is defined as the prior-posterior gain in Shannon information

U(y, \xi) = \int{\log(p(\theta | y, \xi))p(\theta | y, \xi)d\theta} - \int{\log(p(\theta))p(\theta)d\theta} \, .

Lindley (1956) noted that the expected utility will then be coordinate-independent and can be written in two forms

\begin{alignat}{2} U(\xi) & = \int{\int{\log(p(\theta | y,\xi))p(\theta, y | \xi)d\theta}dy} - \int{\log(p(\theta))p(\theta)d\theta} \\ & = \int{\int{\log(p(y | \theta,\xi))p(\theta, y | \xi)dy}d\theta} - \int{\log(p(y| \xi))p(y| \xi)dy} , \end{alignat} \,

of which the latter can be evaluated without the need for evaluating individual posterior PDFs p(\theta | y,\xi) for all possible observations y. Worth noting is that the first term on the second equation line will not depend on the design \xi, as long as the observational uncertainty doesn't. On the other hand, the integral of p(\theta) \log p(\theta) in the first form is constant for all \xi, so if the goal is to choose the design with the highest utility, the term need not be computed at all. Several authors have considered numerical techniques for evaluating and optimizing this criterion, e.g. van den Berg, Curtis & Trampert (2003) and Ryan (2003).

See also

References

Scientific
Method
Treatment
& Blocking
Models
& Inference
Designs:

Completely
Randomized
Glossary · Category · Statistics portal · Statistical outline · Statistical topics